NAS A/TM-20 1 2-2 1733 5/V olume II 
NESC-RP-08-00448 



Probabilistic Requirements (Partial) Verification 
Methods Best Practices Improvement 

Variables Acceptance Sampling Calculators: Empirical 
Testing 


Kenneth L. Johnson/NESC 

Langley Research Center, Hampton, Virginia 

K. Preston White, Jr. 

University of Virginia, Charlottesville, Virginia 


February 2012 


NASA STI Program ... in Profile 


Since its founding, NASA has been dedicated to 
the advancement of aeronautics and space science. 

The NASA scientific and technical information (STI) 
program plays a key part in helping NASA maintain 
this important role. 

The NASA STI program operates under the 
auspices of the Agency Chief Information Officer. It 
collects, organizes, provides for archiving, and 
disseminates NASA’s STI. The NASA STI program 
provides access to the NASA Aeronautics and Space 
Database and its public interface, the NASA Technical 
Report Server, thus providing one of the largest 
collections of aeronautical and space science STI in 
the world. Results are published in both non-NASA 
channels and by NASA in the NASA STI Report 
Series, which includes the following report types: 

• TECHNICAL PUBLICATION. Reports of 
completed research or a major significant phase 
of research that present the results of NASA 
programs and include extensive data or 
theoretical analysis. Includes compilations of 
significant scientific and technical data and 
information deemed to be of continuing 
reference value. NASA counterpart of peer- 
reviewed formal professional papers, but having 
less stringent limitations on manuscript length 
and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 
and technical findings that are preliminary or of 
specialized interest, e.g., quick release reports, 
working papers, and bibliographies that contain 
minimal annotation. Does not contain extensive 
analysis. 

• CONTRACTOR REPORT. Scientific and 
technical findings by NASA-sponsored 
contractors and grantees. 


• CONFERENCE PUBLICATION. Collected 
papers from scientific and technical 
conferences, symposia, seminars, or other 
meetings sponsored or co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 
technical, or historical information from NASA 
programs, projects, and missions, often 
concerned with subjects having substantial 
public interest. 

• TECHNICAL TRANSLATION. English- 
language translations of foreign scientific and 
technical material pertinent to NASA’s mission. 

Specialized services also include creating custom 
thesauri, building customized databases, and 
organizing and publishing research results. 

For more information about the NASA STI 
program, see the following: 

• Access the NASA STI program home page at 
http://www.sti.nasa.sov 

• E-mail your question via the Internet to 
heln@sti.nasa.gov 

• Fax your question to the NASA STI Help Desk 
at 443-757-5803 

• Phone the NASA STI Help Desk at 
443-757-5802 

• Write to: 

NASA STI Help Desk 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 


NAS A/TM-20 1 2-2 1733 5/V olume II 
NESC-RP-08-00448 



Probabilistic Requirements (Partial) Verification 
Methods Best Practices Improvement 

Variables Acceptance Sampling Calculators: Empirical 
Testing 


Kenneth L. Johnson/NESC 

Langley Research Center, Hampton, Virginia 

K. Preston White, Jr. 

University of Virginia, Charlottesville, Virginia 


National Aeronautics and 
Space Administration 

Langley Research Center 
Hampton, Virginia 23681-2199 


February 2012 


Available from: 

NASA Center for AeroSpace Information 
7115 Standard Drive 
Hanover, MD 21076-1320 
443-757-5802 





@ 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

1 of 48 


Volume II 

Probabilistic Requirements (Partial) Verification Methods Best 

Practices Improvement 


Variables Acceptance Sampling Calculators: 
Empirical Testing 


October 27, 2011 


NESC Request No.: TI-08-00448 


1 

a 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

2 of 48 


Approval and Document Revision History 

NOTE: This document was approved at the October 27, 2011, NRB. This document was 
submitted to the NESC Director on January 30, 2012, for configuration control. 


Approved: 

Original Signature on File 

1/30/12 


NESC Director 

Date 


Version 

Description of Revision 

Office of Primary 
Responsibility 

Effective Date 

1.0 

Initial Release 

Kenneth L. Johnson, Statistics 
and Trending, Marshall Space 
Flight Center 

10/27/11 


NESC Request No.: TI-08-00448 



@ 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

3 of 48 


Table of Contents 

Volume II: Technical Assessment Report 


1.0 Notification and Authorization 7 

2.0 Signature Page 8 

3.0 Team List 9 

4.0 Abstract 10 

5.0 Introduction 11 

6.0 Test Protocol 11 

7.0 Results 12 

7.1 Attributes (Binomial) Plan 12 

7.2 Exponential Plan (Lower Limit) 13 

7.3 Exponential Plan (Upper Limit) 16 

7.4 Normal Plan (cr-known) 20 

7.5 Normal Plan (cr-unknown) 23 

7.6 Gamma Plan (Lower Limit) 25 

7.7 Gamma Plan (Upper Limit) 27 

7.8 Weibull Plan (Lower Limit) 29 

7.9 Weibull Plan (Upper Limit) 32 

7.10 Inverse Gaussian 35 

7.11 Pois son (Upper Limit) 41 

8.0 Summary and Conclusions 43 

9.0 Acronyms List 45 

10.0 References 45 

11.0 Appendices 45 


List of Figures 

Ligure 7.1-1. Attributes Simulation Results: Distribution Lunctions and Critical Values 

for X Distributed Binomial (777, p) Under (a) Null Hypothesis po = 0.005 and 
(b) Alternate Hypotheses and pi = 0.001 for an Upper Limit on the Number of 

Nonconforming Observations 12 

Ligure 7.2-1. Exponential Lower Limit Simulation Inputs: Distribution Lunctions and 
Critical Values for X with Lower Limit jc m i n = 1000 Under (a) the Null 


NESC Request No.: TI-08-00448 


@ 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

4 of 48 


Hypothesis po = 0.001 (po = 999,500) and (b) Alternate Hypotheses and 

pi = 0.005 (p,= 199,500) 13 

Figure 7.2-2. Comparison of the Exponential (Lower Limit) and Attributes Sampling OC 

Curves for the Test Case 14 

Ligure 7.2-3. Exponential Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 15 

Ligure 7.3-1. Exponential Upper Limit Simulation Inputs: Distribution Lunctions and 
Critical Values for X with Upper Limit x max = 10,000 and Under (a) Null 
Hypothesis po = 0.001 (po = 1447.65) (b) Alternate Hypotheses and 

pi = 0.005 (//i = 1887.4) 17 

Ligure 7.3-2. Comparison of the Exponential (upper limit) and Attributes Sampling OC 

Curves for the Test Case 18 

Ligure 7.3-3. Exponential Upper Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 19 

Ligure 7.4-1. Normal cr-known, Lower- Limit, Simulation Inputs: Distribution Lunctions 
and Critical Values for X with Lower Limit y m j n = 1000 under (a) the Null 
Hypothesis po = 0.001 (po = 1258) and (b) Alternate Hypotheses and 

pi = 0.005 (pi = 1309) 20 

Ligure 7.4-2. Comparison of the Normal Plan (cr-known) and Attributes Sampling OC 

Curves for the Test Case 21 

Ligure 7.4-3. Normal (cr-known), Lower-Limit, Simulation Results: Sampling Distributions 
for the Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 
Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 
Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 22 


Ligure 7.5-1. Normal (cr-Unknown), Lower-Limit, Simulation Inputs: Distribution 

Lunctions and Critical Values for X with Lower Limit A' mm = 1000 under 
(a) the Null Hypothesis po = 0.001 (po = 1258) and (b) Alternate Hypotheses 


and pi = 0.005 (pi = 1309). Ovals are 99-percent Confidence Bounds on 

the Scatter 24 

Ligure 7.6-1. Gamma Lower- Limit Simulation Inputs: Distribution Lunctions and Critical 

Values for X with Lower Limit x m j n = 1000 under (a) the Null Hypothesis 
po = 0.001 (po = 3377.79) and (b) Alternate Hypotheses and pi = 0.005 

(Pi = 3117.79) 25 

Ligure 7.6-2. Gamma Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A under the (a) Null and (b) Alternate Hypotheses; 


NESC Request No.: TI-08-00448 


@ 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

5 of 48 


Scatterplots of the Sample Statistics under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 26 

Figure 7.7-1. Gamma Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max = 10,000 Under (a) the Null Hypothesis 
Po = 0.001 (//o = 4413.58) and (b) Alternate Hypotheses and p\ = 0.005 

(//, = 5587.57) 27 

Figure 7.7-2. Gamma Upper-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 29 

Figure 7.8-1. Weibull Lower-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Lower Limit x max = 10,000 Under (a) the Null Hypothesis 
po = 0.001 (//o = 1898.10) and (b) Alternate Hypotheses and p\ = 0.005 

(//i = 1712.00) 30 

Figure 7.8-2. Comparison of the Weibull Lower- limit Normal Plan and Attributes 

Sampling OC Curves for the Test Case 31 

Figure 7.8-3. Weibull Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 32 

Figure 7.9-1. Weibull Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max = 10,000 Under (a) the Null Hypothesis 
Po = 0.001 (//o = 7841.64) and (b) Alternate Hypotheses and p\ = 0.005 

(//i = 8121.07) 33 

Figure 7.9-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes 

Sampling OC Curves for the Test Case 34 

Figure 7.9-3. Weibull Upper-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 
Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter 35 


Figure 7.10-1. IG Lower- Limit Simulation Inputs: Distribution Functions and Critical Values 
for X with Upper Limit A' mm = 1000 Under (a) the Null Hypothesis po = 0.001 

(po = 1502.00) and (b) Alternate Hypotheses and p\ = 0.005 (yq = 1413.00) 36 

Figure 7.10-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes 


Sampling OC Curves for the Test Case 37 

Figure 7.10-3. IG Lower-Limit Simulation Results: Sampling Distributions for the Allowable 
Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the 
Sample Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 
99-percent Confidence Bounds on the Scatter 38 


NESC Request No.: TI-08-00448 


1 

a 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

6 of 48 


Figure 7.10-4. IG Lower-Limit Simulation Results Applying a Normal Sampling Plan: 

Sampling Distributions for the Allowable Limit A Under the (a) Null and 

(b) Alternate Hypotheses; Scatterplots of the Sample Statistics Under the 

(c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter 40 

Figure 7.10-5. Comparison of the IG and Normal Distributions with the Same Mean and 

Standard Deviation 41 

Figure 7.11-1. Poisson Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max = 10 Under (a) the Null Hypothesis 

po = 0.001 and (b) Alternate Hypotheses and p\ = 0.005 42 

Figure 7.1 1-2. Poisson Simulation Results: Distribution Functions and Critical Values for X 
Distributed Poisson (27, p) with Limit x max = 10 and Under (a) Null Hypothesis 
Po = 0.001 and (b) Alternate Hypotheses and pi = 0.005 43 

List of Tables 

Table 8.0-1. Summary of Test Results for Continuous Variables Given Lower Limit 

jtmin = 1000 and Nominal OC (po,a) = (0.001,0.2) and (puff) = (0.001,0.1) 44 

Table 8.0-2. Summary of Results for Continuous Variables Given Upper Limit 

*max = 10,000 and Nominal OC (po,a) = (0.001,0.2) and (p\,/3) = (0.001,0.1) 44 

Table 8.0-3. Summary of Results for Discrete Variables 44 


NESC Request No.: TI-08-00448 



@ 

NASA Engineering and Safety Center 
Technical Assessment Report 

Document #: 

NESC-RP- 

08-00448 

Version: 

1.0 

Title: 

Probabilistic Requirements Verification Methods Best 
Practices Improvement 

Page #: 

7 of 48 


Volume II: Technical Assessment Report 

1.0 Notification and Authorization 

The NASA Engineering and Safety Center (NESC) was requested to improve on the Best 
Practices document produced for NESC assessment 07-001-E: Verification of Probabilistic 
Requirements for the Constellation Program (CxP) by giving a recommended procedure for 
using acceptance sampling by variables techniques as an alternative to the potentially resource- 
intensive acceptance sampling by attributes method given in the document. 

An NESC out-of-board activity was approved on January 9, 2008. Mr. Kenneth Johnson at 
Marshall Space Flight Center (MSFC) was selected to lead this assessment. The assessment plan 
was approved at the NESC Review Board (NRB) on March 20, 2008. 
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4.0 Abstract 

Acceptance sampling is a method for verifying lot quality or performance requirements using 
sample data. In this paper, the results of empirical tests intended to assess the accuracy of 
acceptance sampling plan calculators implemented for six variable distributions — binomial, 
exponential, normal, gamma, Weibull, inverse Gaussian (IG), and Poisson are presented. In 
general, these results support the accepted wisdom that variables acceptance plans are superior to 
attributes (binomial) acceptance plans, in the sense that these provide comparable protection 
against both producer’s and consumer’s risk at reduced sampling cost. For the Gaussian and 
Weibull plans, however, there are ranges of the shape parameters for which the required sample 
sizes are in fact larger than the corresponding attributes plans, dramatically so for instances of 
large skew. Tests also confirm that the published IG plan is flawed (White and Johnson, 201 1). 
Appendix A provides the protocol for selecting a sampling plan calculator. Appendix B provides 
a note on hypothesizing an output distribution. 
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5.0 Introduction 

Acceptance sampling by attributes (ASA) assesses the quality of a lot based on the number of 
nonconforming items discovered in a random sample drawn for inspection. Inspection requires 
only a pass/fail determination for each item. Because it is conceptually straightforward, easily to 
implement, and can be applied to qualitative as well as quantitative performance measures, ASA 
is the first choice for sampling inspection. 

For quantitative performance measures, however, a pass/fail determination typically is 
accomplished by comparing the measured value to a limiting value, without regard to magnitude 
of conformance or nonconformance for each item tested. It seems reasonable that this additional 
information might be exploited to decrease the number of items that need to be inspected. This 
is the rationale behind acceptance sampling by variables (ASV). If there is adequate information 
to posit a distribution for the measure, then in many instances the ASV alternative translates into 
significantly smaller samples to achieve the same operating characteristic. While far more 
restrictive in its assumptions, ASV should be considered only when larger samples required by 
ASA is an issue. 

The objective of this paper is to provide an independent assessment of the accuracy of variables 
plans reported in the literature. Note that, with the exception of normal plans, the search for off- 
the-shelf ASV plan calculators was essentially fruitless. This scarcity strongly suggests that non- 
normal ASV largely has been limited to an academic audience and not fully vetted in practice. 
The need to implement and test plans reported in the literature is especially important for those 
plans based on approximations. This paper discusses the results of empirical tests conducted 
using spreadsheet implementation of calculators developed for this purpose. 

6.0 Test Protocol 

The plans reviewed by White and Johnson (2011) were implemented as spreadsheet calculators 
and tested empirically using Monte Carlo simulation. The test protocol enforced a limit standard 
with (1) a specification limit on the measured variable X, either x m \n or x max , as the performance 
indicator I, (2) minimum reliability/? = 0.005, and (3) maximum consumer’s risk /? = 0. 100. 
Additionally, maximum producer’s risk of a= 0.200 was enforced. These particular test 
conditions were chosen as representative of certain high-level requirements in the design of 
spacecraft (White, et al., 2009). 

For each test, values were specified for the limit and for any distribution parameters assumed to 
be known or estimated. The null and alternative means // 0 and ju\ were determined such that, for 
a lower limit, F(x m \ n ;jUo) = 0.001 and F(x m i n ;/q) = 0.005. For an upper limit, 1 -F(x max ;//o) = 
0.999 and 1 - F(x max ,/ui) = 0.995. The corresponding (nj<) sampling plan was then determined 
from the appropriate calculator. 
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One-hundred thousand Monte Carlo trials were run for both the null and alternative distributions, 
each run comprising n observations as determined by the sampling plan. The proportions a and 
ji were estimated from the sampling distribution of the acceptance limit A(n,k). These estimates 
were compared to the specified operating characteristic to assess the accuracy of the plan. The 
efficiency of the variable sampling plan was determined by comparing the required sample size 
to that for the closest attributes sampling plan. 

7.0 Results 

7.1 Attributes (Binomial) Plan 

For the test operating characteristic (OC), the closest attributes sampling plan from the binomial 
calculator is ( n,c ) = (777,1), with null and alternative test points (p 0 , 1 - a) = (0.001,0.8280) and 
(p\,p) = (0.005,0.0998). As shown in Figure 7.1-1, the requirement for integer sample size n 
implies that the specified limits on the risks are enforced, but not necessarily as equalities. This 
plan is otherwise exact, in the sense that (1) the pass/fail determination is necessarily distributed 
binomial ( n,p ) and requires no assumptions regarding the distribution of X and (2) there is a 
single unknown parameter p and additional parameters need not be estimated approximately. 



(a) (b) 

Figure 7.1-1. Attributes Simulation Results: Distribution Functions and Critical Values for X 
Distributed Binomial (777, p) Under (a) Null Hypothesis po = 0.005 and (b) Alternate 
Hypotheses and pi- 0.001 for an Upper Limit on the Number of Nonconforming 

Observations 
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7.2 Exponential Plan (Lower Limit) 

Consider a lower limit of A' nim = 1000 for an exponentially distributed random variable X with 
unknown mean ju. For the test OC, the associated the null and alternative means are 
/Jo = 999,500 and /u\ = 199,500, respectively, as shown in Figure 7.2-1. The variables plan from 
the exponential calculator is ( n,k ) = (2, 2.42722 x 10 3 ). 



Figure 7.2-1. Exponential Lower Limit Simulation Inputs: Distribution Functions and 
Critical Values for X with Lower Limit x,„i„ = 1000 Under (a) the Null Hypothesis po = 0.001 
(fMt = 999,500) and (b) Alternate Hypotheses and pi = 0.005 ( pi = 199,500) 


Figure 7.2-2 compares the OC curves for the exponential and attribute plans. These curves are 
similar, with the exponential sampling plan modestly more conservative with respect to risks. 
Note that here and throughout, the OC curves are theoretical and derived from the plans, and 
may or may not agree with empirical results. 
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p 

Figure 7.2-2. Comparison of the Exponential ( Lower Limit) and Attributes Sampling OC 

Curves for the Test Case 

Simulations were run to estimate the distribution of the test statistic, the allowable limit A = kju , 
where the estimated mean u is the sample mean X ( n ) and n an k are given by the sampling 

plan. Figure 7.2-3(a) and (b) shows the sample distributions of A for the null and alternate 
hypotheses, respectively. From these distributions, it was observed that there was no practical 
difference between the estimated risks of a - 0.200 and f3 = 0.082 and those predicted by the 
OC curve. 

Figure 7.2-3(c) and (d) are scatterplots of the sample mean and standard deviation for each of the 
100,000 simulation trials for the null and alternate hypotheses, respectively. Also shown on each 
scatterplot is the threshold value of A, the constant p = x m \J k = 41 1,993. This line divides each 
sample into two sets, with the conforming observations above the line and nonconforming 
below. 
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Figure 7.2-3. Exponential Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 


Each figure shows that the proportion of nonconforming observations subtended by the line is 
approximately 20 percent for the null distribution and the proportion of conforming observations 
subtended by the line is approximately 8.20 percent for the alternate distribution. Increasing the 
limiting value x m i n will decrease the proportion conforming (increasing a and decreasing f:S). 
Increasing the value of k will have a similar effect. 
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The respective 99-percent confidence bounds for each data set are shown as ovals on the 
scatterplots. The area subtended by these bounds depends on the number runs n. Increasing the 
number of runs will decrease the radius of the confidence bound, decreasing a and for fixed/?. 

The lower-limit exponential plan clearly outperforms the attributes plan, by providing equivalent 
risk protection at a dramatically reduced sampling cost. The number of simulations indicated by 
the binomial is n = 111, whereas the variables plans require only n = 2, which is 0.26 percent of 
the computational effort of the attributes plan. From Figure 7.2-1, it should be clear that this 
economy was achieved because the null and alternate distributions are distinct — observations 
that are typical of one distribution are rare for the other. The economy was further enhanced by 
the need to estimate only a single parameter sample from the data. 

It seems obvious, however, that two observations were inadequate to determine the exponential 
distributional form if it is otherwise unknown. Thus, in the case of exponentially distributed 
measurement, it appears that the number of trials required in many cases will be dictated by the 
need to determine the form of the output distribution, rather than the need to satisfy a limit 
standard. 

7.3 Exponential Plan (Upper Limit) 

Consider an upper limit of x max = 10,000 for an exponentially distributed random variable X with 
unknown mean ju. For the test OC, the associated the null and alternative means are 
ju 0 = 1447.65 and /q = 1887.4, respectively, as shown in Figure 7.3-1. The variables plan from 
the exponential calculator is (n,k) = (66,6.26922). 
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Figure 7.3-1. Exponential Upper Limit Simulation Inputs: Distribution Functions and 
Critical Values for X with Upper Limit x max = 10,000 and Under (a) Null Hypothesis 
po = 0.001 (juo = 1447.65) (b) Alternate Hypotheses and pi - 0.005 (jui = 1887.4) 


Figure 7.3-2 compares the OC curves for the exponential upper limit and attributes plans. These 
curves again are similar and almost identical at the OC points tested. Simulations were run to 
generate the distribution of the test statistic A = kfi for the exponential tests. Results are shown 

in Figure 7.3-3. There is no practical difference between the estimated risks of a = 0.200 and 
f = 0.082 and those shown on the OC curve. 


The exponential upper-limit plan also outperforms the attributes plan, by providing equivalent or 
superior risk protection at a substantially reduced sampling cost. The number of simulations 
indicated by the binomial is n = 111, whereas the variables plans require n = 66, which is 
8.49 percent of the computational effort of the attributes plan. From Figures 7.3-1 and 7.3-3, it is 
clear that the null and alternate distributions are not nearly as distinct as in the case of a lower 
bound, owning to the skew. Thus, the economy is not as great as that for a lower bound, but is 
still impressive. Again, it may be that 66 observations are inadequate to determine the 
exponential distributional form and that the number of trials required in many cases will be 
dictated by this need, rather than the need to satisfy a limit standard. 
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Figure 7.3-2. Comparison of the Exponential ( upper limit) and Attributes Sampling OC 

Curves for the Test Case 
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Figure 7.3-3. Exponential Upper Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 
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7.4 Normal Plan (cr-known) 

Consider a lower limit of Xmin = 1000 for a normally distributed random variable X with unknown 
mean // and known standard deviation cr= 100. For the test OC, the associated the null and 
alternative means are /u 0 = 1309 and jui = 1258, respectively, as shown in Figure 7.4-1. The 
variables plan from the norm calculator is ( n,k ) = (18,2.88632) is also shown. 



(a) (b) 

Figure 7.4-1. Normal cr-known , Lower-Limit, Simulation Inputs: Distribution Functions and 
Critical Values for X with Lower Limit x,„i n = 1000 under (a) the Null Hypothesis po = 0.001 
(juo = 1258) and (b) Alternate Hypotheses and pi = 0.005 (jui = 1309) 


Figure 7.4-2 compares the OC curves for the normal and attributes plans. These curves again are 
similar, with the normal plan modestly more conservative in the region tested. Simulations were 
run to generate the distribution of the test statistic A 0 = ju — 100k for the normal tests. Results 
are shown in Figure 7.4-3. There was no practical difference between estimated risks of 
a = 0.193 and (3 = 0.096 and those predicted by the OC curve. 


The normal plan outperformed the attributes plan, by providing equivalent or superior risk 
protection at a dramatically reduced sampling cost. The number of simulations indicated by the 
binomial was n = 111, whereas the variables plans require only n = 18, which is 2.32 percent of 
the computational effort of the attributes plan. Given the symmetry of the normal distribution, 
the same results for an upper bound were expected. 
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Figure 7.4-2. Comparison of the Normal Plan ( o-known) and Attributes Sampling OC Curves 

for the Test Case 
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(C) (d) 

Figure 7.4-3. Normal ( cr-known), Lower-Limit, Simulation Results: Sampling Distributions 
for the Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses ; Scatterplots of the 
Sample Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent 

Confidence Bounds on the Scatter. 
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7.5 Normal Plan (cr-unknown) 

Test conditions are identical to the cr-known case, except that standard deviation is now <r , and 
estimated from the data for each trial. The variables plan from the normal calculator is 
(n,k) = (88,2.88632) with OC curve identical to that for the cr-known case. 

Simulations were run to generate the distribution of the test statistic A = // - 1 00k for the normal 
tests. Results are shown in Figure 7.5-1. The estimated risks of a - 0.191 and /? = 0.109 are not 
quite those and predicted by the OC curve, with modestly lower producer’s risk and modestly 
higher consumer’s risk. The normal plan outperforms the attributes plan, by providing 
equivalent or protection at a dramatically reduced sampling cost. The number of simulations 
indicated by the binomial was n = 111, whereas the variables plans require only n = 88, which is 
1 1.3 percent of the computational effort of the attributes plan. Given the symmetry of the normal 
distribution, the same results for an upper bound was expected. 
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Figure 7.5-1. Normal ( a-Unknown), Lower-Limit, Simulation Inputs: Distribution Functions 
and Critical Values for X with Lower Limit j = 1000 under (a) the Null Hypothesis 
Po = 0.001 (/Jo = 1258) and (b) Alternate Hypotheses andpi = 0.005 (/// = 1309). Ovals are 

99-percent Confidence Bounds on the Scatter 
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7.6 Gamma Plan (Lower Limit) 

Consider a lower limit of x m j n = 1000 for a random variable X distributed gamma with unknown 
shift parameter 8 and estimated shape and scale parameters, v = 10 and X = 337.779 . For the test 
OC, the associated null and alternative means were jl/q = 3377.79 and ju\ = 3140.79, as shown in 
Figure 7.6-1. The variables plan from the gamma calculator was (n,k) = (206,2.13126). 
Simulations were run to generate the distribution of the test statistic Aq = ju-100k . Results are 
shown in Figure 7.6-2. There was no practical difference between estimated risks of 
a = 0.191 and /) = 0.096 and those prescribed. The gamma plan outperforms the attributes 
plan, by providing equivalent or superior risk protection at a reduced sampling cost. The number 
of simulations indicated by the binomial was n = 777, whereas the variables plans require 
n = 206, which is 25.5 percent of the computational effort of the attributes plan. 




Figure 7.6-1. Gamma Lower-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Lower Limit x m i„ = 1000 under (a) the Null Hypothesis po = 0.001 
(/Jo = 3377.79) and (b) Alternate Hypotheses and pi - 0.005 (/// = 3117.79) 
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Figure 7.6-2. Gamma Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 
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Note, however, for the test OC and v < 3.431 , the variables plan required samples larger than the 
attributes plan. For example, v = 0.200 sampling plan from the gamma calculator was 
(n,k) = (1960,1.46100) — nearly three times larger than the attributes plan. Given the other 
advantages of attributes sampling, gamma plans would never be used under this condition. This 
contradicts the accepted wisdom that ASV will always be more efficient than ASA. The 
operative point was that the sample size for any variables plan should be compared to the 
corresponding attributes plan and the benefit of the potential computational savings achieved 
(if any) assessed in the light of the restrictive assumptions imposed by variables sampling. 

7.7 Gamma Plan (Upper Limit) 

Consider an upper limit of x max = 10,000 for a random variable X distributed gamma with 
unknown shift parameter 8 and estimated shape and scale parameters, v = 10 and 
X = 441.358 . For the test OC, the associated the null and alternative means are jl/q = 4413.58 and 
jui = 5587.57, respectively, as shown in Figure 7.7-1. The variables plan from the gamma 
calculator is (n,k) = (77,3.66693). 


100 ... 



100 ... 



Figure 7. 7-1. Gamma Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max = 10,000 Under (a) the Null Hypothesis po = 0.001 
(juo = 4413.58) and (b) Alternate Hypotheses and pi = 0.005 (jui = 5587.57) 


Simulations were run to generate the distribution of the test statistic A 0 = ju + 1 00k for the 
Gamma test. Results are shown in Figure 7.7-2. The estimated risks of a = 0.186 and 
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/3 = 0.104 are not quite those and predicted by the OC curve, with modestly lower producer’s 
risk and modestly higher consumer’s risk. 

The gamma plan outperforms the attributes plan, by providing equivalent or superior risk 
protection at a reduced sampling cost. The number of simulations indicated by the binomial is 
n = 777, whereas the variables plans require n = 77, which is 9.91 percent of the computational 
effort of the attributes plan. 
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(c) (d) 

Figure 7.7-2. Gamma Upper-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 


7.8 Weibull Plan (Lower Limit) 

Consider a lower limit of x m i n = 1000 for a random variable X distributed Weibull with unknown 
shift parameter 8 and estimated shape and scale parameters, v = 1 0 and A = 1995.16 . For the test 
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OC, the associated the null and alternative means are /j 0 = 1898.10 and //| = 1712.00, 
respectively, as shown in Figure 7.8-1. The variables plan from the Weibull calculator is 
(n,k) = (91,3.62929. Figure 7.8-2 compares the OC curves for the normal and attributes plans. 
These curves again are similar. 


- 10 ... - 10 ... 




(a) (b) 

Figure 7.8-1. Weibull Lower-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Lower Limit x max - 10,000 Under (a) the Null Hypothesis po = 0.001 
(po = 1898.10) and (b) Alternate Hypotheses and pi = 0.005 (pi = 1712.00) 
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— — Wiebull Lower Limit — — Attributes 

Figure 7.8-2. Comparison of the Weibull Lower-limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case 

Simulations were run to generate the distribution of the test statistic A i) = ju-kc t for the Weibull 
test. Results are shown in Figure 7.8-3. The estimated risks of a = 0.1889 and 
p = 0.0794 were modestly more conservative than those predicted by the OC curve and there is 
no practical difference between the estimated risks and those predicted by the OC curve. The 
Weibull plan outperforms the attributes plan, by providing equivalent or superior risk protection 
at a reduced sampling cost. The number of simulations indicated by the Binomial n = 111 , 
whereas the variables plans require n = 91, which is 11.7 percent of the computational effort of 
the attributes plan. 

Note that for the test OC and v < 2.157, the variables plan requires samples larger than the 
attributes plan and consequently would never be used. For example, v — 2 and all else being 
equal, the sampling plan from the Weibull calculator is (n,k) = (1071,1.81126). This contradicts 
the accepted wisdom that ASV will always be more efficient than ASA. 
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Figure 7.8-3. Weibull Lower-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 


7.9 Weibull Plan (Upper Limit) 

Consider an upper limit of v max = 10,000 for a random variable X distributed gamma with 
unknown shift parameter 8 and estimated shape and scale parameters, v = 10 and X = 3800.00 . 
For the test OC, the associated the null and alternative means are jUq = 7841.64 and p\ = 8121.07, 
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respectively, as shown in Figure 7.9-1. The variables plan from the gamma calculator was 
(n,k) = (156,2.17779). 
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Figure 7.9-1. Weibull Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max - 10,000 Under (a) the Null Hypothesis po = 0.001 
(juo = 7841.64) and (b) Alternate Hypotheses and pi = 0.005 (jui = 8121.07) 

Figure 7.9-2 compares the OC curves for the normal and attributes plans. These curves again are 
very similar, with the normal plan modestly more conservative in the region tested. Simulations 
were run to generate the distribution of the test statistic \ = ju - 1 00k for the Weibull test. 

Results are shown in Figure 7.9-3. The estimated risks of a = 0.1883 and [1 = 0.0806 were 
modestly more conservative than those predicted by the OC curve. 

The Weibull plan outperformed the attributes plan, by providing equivalent or superior risk 
protection at a reduced sampling cost. The number of simulations indicated by the Binomial 
n = 111 , whereas the variables plans require n = 156, which is 20.1 percent of the computational 
effort of the attributes plan. Note that for the test OC and v< 0.4069, the variables plan requires 
samples larger than the attributes plan and consequently would never be used. For example, 
v = 0.200 sampling plan from the gamma calculator is (n,k) = (4368,9.43402) — more than five 
times larger than the attributes plan. 
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Figure 7.9-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case 
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Figure 7.9-3. Weibull Upper-Limit Simulation Results: Sampling Distributions for the 
Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 
Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 


7.10 Inverse Gaussian 

Consider a lower limit of v rnax = 1000 for a random variable X distributed Poisson with unknown 
shift parameter 8 and estimated mean ju = 1502 and shape parameter A = 100,000. For the test 
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OC, the associated null and alternative means were //<) = 1502.00 and ju\ = 1413.00, respectively, 
as shown in Figure 7.10-1. The variables plan from the IG calculator was ( n,k ) = (18,2.88632). 



(a) (b) 

Figure 7.10-1. IG Lower-Limit Simulation Inputs: Distribution Functions and Critical Values 
for X with Upper Limit x = 1000 Under (a) the Null Hypothesis po = 0.001 (po = 1502.00) 
and (b) Alternate Hypotheses and pi = 0.005 (/// = 1413.00) 

Figure 7.10-2 compares the OC curves for the normal and attributes plans. These curves were 
similar, with the normal plan modestly more conservative in the region tested. Note, however, 
that the theoretical OC curve for the published plan did not match the empirical data, as shown 
below. 
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Figure 7.10-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case 

The test statistic for the IG plan (Aminzadeh, 1996) is: 

A = fi{\ + k^jx ^ n /T j = /} (l + Q(k, , T) > x nii „ . 

Simulations were run to generate the distribution of the statistic and the results are shown in 
Figure 7.10-3. The estimated producer’s risk of a = 0.173 was modestly conservative. 

However, the estimated consumer’s risk of /3 = 0.382 was unacceptably large — almost four 
times that specified — and far from that predicted for the theoretical OC. This confirms the error 
in derivation reported by White and Johnson (2011). The scatter diagrams plot the value of the 
sample mean ju against the parameter group Q for all 100,000 simulation trials. It was apparent 
that the 99-percent confidence bounds on the simulation data are too large and that the 
(n,k) = (18,2.88632) sampling plan is too small. 
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Figure 7.10-3. IG Lower-Limit Simulation Results: Sampling Distributions for the Allowable 
Limit A Under the (a) Null and (b) Alternate Hypotheses ; Scatterplots of the Sample Statistics 
Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence Bounds on 

the Scatter. 


As a check, second simulation experiment was run using the same IG distribution, but this time 
following the normal sampling plan (n,k) = (88,2.88632) and corresponding normal test statistic. 
Results are shown in Figure 7.10-4. The larger sample size has the desired effect of compacting 
the confidence bounds. However, the estimated producer’s risk of a = 0.743 was exceedingly 
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large — almost four times that specified — while the estimated consumer’s risk of /? = 0.020 was 
highly conservative. The reason for this is clear from Figure 7.10-5, which compares the IG and 
normal density functions with the same mean and standard deviation. Even though the skew in 
the IG distribution is modest, the lower tail of the normal distribution is considerably fatter than 
that of the IG. Therefore, the normal plan is conservative with respect to consumer’s risk for IG 
data unless these data are suitably transformed. 
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Figure 7.10-4. IG Lower-Limit Simulation Results Applying a Normal Sampling Plan: 
Sampling Distributions for the Allowable Limit A Under the (a) Null and (b) Alternate 
Hypotheses; Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 
Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. 
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Figure 7.10-5. Comparison of the IG and Normal Distributions with the Same Mean and 

Standard Deviation 


Figure 7.10-3 shows the critical value of the test statistic for the normal plan, k = 2.88632, as a 
solid line. The effect of reducing this value to k = 2.60000 as a dashed line is also shown. The 
(n,k) = (88,2.60000), while approximate, was much closer to the desired plan for the test OC than 
either the normal or published IG plans. 


7.11 Poisson (Upper Limit) 

Consider an upper limit of x max =10 for a random variable X distributed Poisson with unknown 
mean //. For the test OC, the associated null and alternative means were po =2.96051 and 
//i =3.79612, respectively, as shown in Figure 7.11-1. The variables plan from the gamma 
calculator was ( n,c ) = (27,88). 
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(a) (b) 

Figure 7.11-1. Poisson Upper-Limit Simulation Inputs: Distribution Functions and Critical 
Values for X with Upper Limit x max = 10 Under (a) the Null Hypothesis po = 0.001 and 

(b) Alternate Hypotheses and pi - 0.005 


Simulations were run to generate the distribution of the test statistic Y = '^ J "_ Q X j . Results are 

shown in Figure 7.11-2. There was no practical difference between the estimated risks of 
estimated risks of a = 0.192 and /? = 0.097 and those specified by the OC. 
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(a) (b) 

Figure 7.11-2. Poisson Simulation Results: Distribution Functions and Critical Values for X 
Distributed Poisson (27, p) with Limit x max = 10 and Under (a) Null Hypothesis po = 0.001 and 

(b) Alternate Hypotheses and p t - 0.005 


The Poisson plan dramatically outperformed the attributes plan, by providing equivalent or 
superior risk protection at a reduced sampling cost. The number of simulations indicated by the 
binomial was n = 27, whereas the variables plans require n = 111, which is 0.347 percent of the 
computational effort of the attributes plan. 


8.0 Summary and Conclusions 

This paper presents the results of empirical tests designed to provide an independent assessment 
of the validity and accuracy of six published ASV procedures. These results are summarized in 
Tables 8.0-1 through 8.0-3. Overall, the plans were shown to provide adequate or superior 
protection against producer’s and consumer’s risks for samples substantially smaller than those 
required for the corresponding ASA plans. The exception is the IG plan, which was previously 
shown to be in error. 
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Table 8.0-1. Summary of Test Results for Continuous Variables Given Lower Limit x min = 1000 
and Nominal OC (po,a) = (0.001,0.2) and (pi,P) = (0.001,0.1) 


Variable 

n 

k 

a 

P 

n \/n a 

Exponential!//) 

2 

2.43 x 
10“ 3 

0.200 

0.082 

0.003 

Normal(//, cr = 100) 

18 

2.886 

0.191 

0.097 

0.023 

Normal!// & ) 

88 

2.886 

0.191 

0.097 

0.099 

Gamma( v = 10 , i = 338 ,S) 

206 

2.131 

0.193 

0.096 

0.224 

Weibull! v = 10, i = 1995 ,5) 

91 

3.623 

0.189 

0.079 

0.117 

IG(// = 1502, i = 100,000, S) 

18 

2.886 

0.173 

0.382 

unusable 


Table 8.0-2. Summary of Results for Continuous Variables Given Upper Limit x mai = 10,000 and 
Nominal OC (po,a) = (0.001,0.2) and (pi,P) = (0.001,0.1) 


Variable 

n 

k 

a 

P 

n J n a 

Exponential!//) 

66 

6.26922 

0.200 

0.082 

0.085 

Gamma( v = 10 , A = 441 ,5) 

77 

3.667 

0.189 

0.104 

0.099 

Weibull! v = 10, i = 3800 ,5) 

156 

3.623 

0.188 

0.081 

0.201 


Table 8.0-3. Summary of Results for Discrete Variables 


Variable 

n 

c 

a 

b 

// \/ll a 

Binomial ( n,p ) 

777 

1 

0.188 

0.100 

1 

Poisson (, n,p ) 

21 

88 

0.191 

0.097 

0.035 


While this overall conclusion supports the assertion in the literature that ASV plans require 
smaller samples than ASA plans, it was also discovered that this assertion does not hold 
absolute. In particular, for gamma and Weibull variables with small shape parameters, the ASV 
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plans were in fact larger than the corresponding ASV plans. It is believed that this discovery is 
original and needs to be assimilated into the literature on acceptance sampling. 

9.0 Acronyms List 

ASA Acceptance Sampling by Attributes 

ASV Acceptance Sampling by Variables 

ATK Alliant Techsystems, Inc. 

CxP Constellation Program 

IG Inverse Gaussian 

LaRC Langley Research Center 

MSFC Marshall Space Flight Center 

MTSO Management Technical Support Office 

NESC NASA Engineering and Safety Center 

NRB NESC Review Board 

OC Operating Characteristic 
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11.0 Appendices 

Appendix A. Protocol for Selecting a Sampling Plan Calculator 
Appendix B. Note on Hypothesizing an Output Distribution 
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Appendix A. Protocol for Selecting a Sampling Plan Calculator 
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Appendix B. Note on Hypothesizing an Output Distribution 

The sampling plan calculators can be used to determine the number of simulation replications or 
runs needed, i.e., the number of Monte Carlo trials yielding the sample size n specified by the 
plan for a given requirement. However, the choice of which calculator to use depends on 
knowing the form of the distribution of the verification output. The size of sampling plans for 
different distributions can vary by orders of magnitude depending on this distribution. 

Pass/fail output always have a binomial distribution and therefore the n can be determined using 
the binomial (attributes) calculator before any simulation runs are made. For variables output, 
the choice may not be as obvious. 

If historical or test data exist for variables outputs that are thought to be similar in nature to the 
expected verification output, a verifier may be able to make use of such prior knowledge to guess 
what form the distribution might assume. S/he needn’t be concerned about the values of the 
distribution parameters at this step, just with the general shape of the distribution. The choice of 
distribution (hypothesis) and parameter estimates can be checked once data are generated for the 
verification analysis. 

There also are theoretical considerations that might suggest one or more distributions, or at least 
rule out some of the distributions available. For example, the Normal distribution often is a 
reasonable model for errors of various types, where the magnitude of the error is equally likely to 
be positive or negative and smaller errors are more common than larger errors. By virtue of the 
central limit theorem, the Normal distribution also is often a reasonable model for the sum of a 
large number of other random quantities, no matter how these quantities are distributed. Because 
the Normal distribution has a negative tail, however, it may not be a reasonable model for a time, 
or a count, or some other output that must be strictly positive. 

Because of the “memoryless property”, the exponential distribution is a suitable model for the 
time between the occurrence of events, if these occurrences have a constant rate and the numbers 
of occurrence in disjoint time periods are independent (i.e., the time of occurrence of one event 
does not influence the time of occurrence of subsequent events). Events might be the failure of a 
component or system, or the arrival of a customer needing service, or the placing of orders 
against an inventory. By the same property, the number of events occurring during any fixed 
period of time under these same assumptions is an integer count distributed Poisson. 

The gamma and Weibull distributions frequently are used to model the time to some event, where 
the event might be a failure or the completion of some task, such as a service or repair. The 
Weibull is quite flexible in application, can have positive or negative skew, and frequently is 
used as the assumed distribution when modeling in the absence of data. 

The lognormal distribution (the normal distribution applied to the natural logarithm of the data) 
has applications similar to the gamma and Weibull. It takes on shapes similar to the gamma and 
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Weibull with positive skew, but can have a large “spike” at the mode close to zero. There are 
other transformations to the Normal that may also be appropriate. 

Before constructing a hypothesis and collecting any data, it is a good idea to consult a statistician 
to help in this activity. It is also important to be aware that the data, once generated, may result 
in a finding that the hypothesized distribution is not the conservative choice. The analyst, 
managers and other verifiers should have a contingency plan covering the possibility that the 
simulation trials already run are not sufficient for verification at the confidence level required. 
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